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FOREWORD 


The  Individual  Combatant  Simulation  System  (ICSS)  concept  was  originally  derived  from  the 
independent  research  and  development  efforts  of  the  Air  Force  Armstrong  Laboratory  Human 
Resources  Directorate  (AL/HR),  the  U.S.  Army  Research  Laboratory  Human  Research  and 
Engineering  Directorate  (ARL/HRED),  the  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI),  the  Marine  Corps  Systems  Command  (MARCORSYSCOM),  and  the 
Naval  Air  Warfare  Center  Training  Systems  Division  (NAWC-TSD).  In  1993,  each  of  these 
organizations  had  research  and  development  programs  under  way  to  improve  various  aspects  of 
simulation  for  the  individual  combatant,  whether  fighter  or  maintainer.  In  June  1994,  with  the 
encouragement  of  the  Defense  Modeling  and  Simulation  Office,  these  organizations  entered  into  a 
cooperative  agreement  for  joint  conduct  of  the  ICSS  program. 

The  objectives  of  the  ICSS  program  were  to;  insert  the  individual  combatant  into  the 
Distributed  Interactive  Simulation  (DIS)  compliant  virtual  environment;  develop  more  accurate 
representations  of  hostile  combatants,  neutrals,  and  friendly  forces  in  a  dynamic  synthetic 
environment;  and  develop  a  more  realistic  human  interface.  ICSS  will  enhance  the  ability  of  the 
Department  of  Defense  to  train  individual  combatant  and  leader  skills;  conduct  virtual  prototyping 
of  developmental  items;  and  provide  the  ability  to  conduct  development  and  analysis  of  system 
utility,  maintainability,  and  human  centered  design. 

This  report  presents  a  plan  for  the  conduct  of  the  assessment  of  the  ICSS  program.  The  plan 
was  presented  to  the  ICSS  program  manager  in  January  1996.  While  the  ICSS  program  was 
terminated  prior  to  the  conduct  of  the  evaluation,  the  plan  nevertheless  provides  an  approach  to 
the  evaluation  of  behavioral  issues  in  virtual  simulations  that  should  be  of  valued  to  other 
researchers  and  program  managers. 


ZITAM.  SIMUTIS 
Deputy  Director 
(Science  and  Technology) 


EDGAR  M.  JOHNSON 
Director 
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INDIVIDUAL  COMBATANT  SIMULATION  SYSTEM  (ICSS)  ASSESSMENT  PLAN 

Introduction 

Purpose 

The  purpose  of  this  document  is  to  describe  the  assessment  plan  for  the  Individual  Combatant 
Simulation  System  (ICSS)  at  a  level  of  detail  that  makes  it  possible:  to  obtain  consensus  among 
program  participants  regarding  assessment  issues,  approaches,  and  tasks;  and  to  estimate  the  cost 
of  the  conduct  of  the  plan. 


Organization  of  the  Plan 

The  background  section  describes  the  objectives  of  the  ICSS  program,  its  components,  and  the 
goals  of  the  ICSS  assessment.  The  Scope  of  the  Assessment,  in  terms  of  the  ICSS  tasks  which 
are  included,  time  constraints,  and  assessment  approaches  are  presented.  The  approach  for  the 
development  of  lessons  learned,  which  applies  to  all  ICSS  tasks,  is  then  described.  Each  ICSS 
task  (and  where  appropriate,  subtasks)  is  then  described  along  with  the  assessment  issues, 
general  assessment  approach,  type  of  approach,  scenario,  performance  measures,  and  tasks 
necessary  to  conduct  the  assessment.  The  computer  resources  and  requirements  and  resources  of 
the  participating  organizations  are  then  presented.  Finally,  a  labor  and  cost  estimate  is  then 
provided  for  each  task.  . 


Background 

The  ICSS  concept  was  derived  from  the  originally  independent  research  and  development 
efforts  of  the  Air  Force  Armstrong  Laboratory  Human  Resources  Directorate  (AL/HR),  the  U.S. 
Army  Research  Laboratory  Human  Research  and  Engineering  Directorate  (ARL-HRED),  the 
U.S.  Army  Research  Institute  (ARI),  the  Marine  Corps  Systems  Command 
(MARCORSYSCOM),  and  the  Naval  Air  Warfare  Center  Training  Systems  Division  (NAWC- 
TSD).  In  1993,  each  of  these  organizations  had  research  and  development  programs  under  way 
to  improve  various  aspects  of  simulation  for  the  individual  combatant,  whether  fighter  or 
maintainer.  MARCORSYSCOM  sponsored  NAWC-TSD's  development  of  the  Team  Target 
Engagement  Simulator  (TTES),  a  squad-level  tactical  and  marksmanship  trainer  based  on  the  use 
of  computer-generated  environment.'  Both  AL/HR  and  ARL-HRED  were  sponsoring 
improvements  in  the  use  of  articulated  human  figure  models  for  concept  development  and  system 
design.  ARL-HRED  had  also  initiated  work  to  develop  the  Individual  Soldier  Mobility  System 
(ISMS),  a  more  natural  human  interface  with  simulated  environments.  ARI  had  a  program  of 
research  under  way  to  improve  the  use  of  virtual  environments  for  individual  combatant  training, 
mission  planning,  and  mission  rehearsal.  In  early  1994,  with  the  encouragement  of  the  Defense 
Modeling  and  Simulation  Office,  these  organizations  entered  into  a  cooperative  agreement  for 
joint  conduct  of  the  ICSS  program. 


^At  the  current  state  of  its  development,  TTES  is  limited  to  the  simultaneous  insertion  of 
two  individuals. 
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The  objectives  of  the  ICSS  program  are  to:  insert  the  individual  combatant  into  the 
Distributed  Interactive  Simulation  (DIS)  compliant  virtual  environment;  develop  a  more  accurate 
representation  of  hostile  combatants,  neutrals  and  friendlies  in  a  dynamic  synthetic  environment; 
and  develop  a  more  realistic  human  interface.  ICSS  will  enhance  the  ability  of  DoD  to  train 
individual  combatant  and  leader  skills,  conduct  virtual  prototyping  of  developmental  items,  and 
provide  the  ability  to  conduct  development  and  analysis  of  system  utility,  maintainability  and 
human  centered  design. 

The  ICSS  program  was  initially  proposed  as  a  series  of  eight  tasks  of  varying  duration  to  be 
carried  out  during  a  two  year  period,  with  some  tasks  beginning  in  year  one  and  some  in  year 
two.  Each  task  had  a  government  task  leader,  responsible  for  its  conduct,  and  most  tasks  had  a 
performing  contractor.  Tasks,  task  leaders,  and  performing  contractors  are  shown  in  Table  1. 

The  focus  of  the  tasks  was  on  the  development  of  new  capabilities,  not  their  integration. 
Consequently,  there  is  not  a  single  ICSS.  There  are  instead  a  number  of  different  capabilities  and 
products,  in  varying  stages  of  development,  which  could  potentially  be  integrated  but  which  must 
be  assessed  independently. 


Approach  to  the  ICSS  Assessment 
Goals 


The  overall  goals  of  the  ICSS  assessment  are  to: 

Demonstrate  and  document,  both  qualitatively  and  quantitatively,  the  extent  to 
which  ICSS  technologies  can  enhance  the  training  of  the  individual  combatant 
and  the  development  of  doctrine,  tactics,  and  systems  for  individual  combatants. 

Identify  areas  of  ICSS  technology  where  future  development  is  required. 

Document  for  future  developers  the  "lessons  learned"  in  the  conduct  of  the 
component  tasks. 
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Table  1 


less  Tasks.  Task  Leaders,  and  Contractors 


Task  1 :  DIS  CONTROL  OF  ARTICULATED  HUMAN  FIGURE  BEHAVIORS 

Task  Leader:  AL/HR 

Contractor:  University  of  Pennsylvania _ 

Task  2:  LINKING  INDIVIDUAL  COMBATANTS  WITH  COMPUTER 

CONTROLLED  REPRESENTATIONS  OF  SQUAD/FIRE  TEAM 
MEMBERS 
Task  Leader:  ARI 

Contractor:  University  of  Central  Florida  Institute  for  Simulation  and  Training _ 

Task  3 :  SELECTED  ACTIONS  OF  INDIVIDUAL  NEUTRALS  AND  HOSTILE 

COMBATANTS 
Task  Leader:  NAWC-TSD 

Contractors:  Naval  Postgraduate  School _ 

Task  4:  DIS  COMPUTER  CONTROLLED  HOSTILE  COMBATANTS  (CCHC) 

Task  Leader:  NAWC-TSD 

Contractor:  University  of  Central  Florida  Institute  for  Simulation  and  Training _ 

Task  5 :  DIS  CATALOGING  AND  AUDITING 

Task  Leader:  ARI 

Contractor:  University  of  Central  Florida  Institute  for  Simulation  and  Training _ 

Task  6:  HUMAN  INTERFACE  FOR  DIS 

Task  Leader:  ARL-HRED 

Contractor:  Sarcos  Research  Corporation _ 

Task  7:  INDIVIDUAL  COMBATANT  HARDWARE  SUITE 

Task  Leader:  Unfunded 

Contractor:  Unfunded _ 

Task  8:  ICSS  ASSESSMENT 

Task  Leader:  ARI 

Contractor:  To  be  Determined 
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Scope 


The  following  ICSS  tasks  will  be  included  in  the  assessment.  Tasks  numbers  are  taken  from 
the  less  Program  Development  Plan  (PDP),  dated  6  April  1994. 

Task  1.  PIS  Control  of  Articulated  Human  Figure  Behaviors.  This  work  will  develop  and 
demonstrate  the  capability  of  using  DIS  to  drive  graphic  representations  of  human  figures 
through  Jack,  a  software  tool  for  manipulating  and  displaying  articulated  human  3-D  forms.  The 
task  has  two  major  subtasks;  one  involves  the  use  of  Jack  in  combat,  and  the  other  in 
maintenance. 

Task  2.  Linking  Individual  Combatants  with  Computer  Controlled  Representations  of 
Squad/Fire  Team  Members.  The  purpose  of  this  task  is  to  develop  the  voice  recognition  and 
limb  tracking  technologies  to  control  dismounted  Semi-Automated  Forces  (SAFOR)  combatants 
through  verbal  commands  or  hand  and  arm  signals. 

Task  3.  Selected  Actions  of  Individual  Neutrals  and  Hostile  Combatants.  This  task  will 
identify  and  define  techniques,  tactics  and  behaviors  employed  by  hostile  combatants  and 
neutrals  and  support  the  development  of  computer  controlled  hostile  combatants  and  neutrals 
(CCHC/N)  for  virtual  environments  and  DIS.  Since  the  outcomes  of  this  tasks  will  be 
incorporated  directly  into  Task  4,  they  will  be  assessed  as  part  of  Task  4. 

Task  4.  DIS  Computer  Controlled  Hostile  Combatants  rCCHCl  This  task  will  develop  and 
implement  computer  controlled  representations  of  hostile  combatants  to  interact  with  the 
dismounted  DIS  trainee. 

Task  5.  DIS  Cataloging  And  Auditing.  This  task  will  analyze  the  adequacy  of  DIS  database 
requirements  and  networking  protocols  for  the  dismounted  combatant  in  virtual  environments. 

Task  6.  Human  Interface  for  DIS.  This  task  will  develop  a  cost-effective  mobility  simulator 
that  can  be  used  for  Virtual  Battlefield  Training  by  the  Battle  Labs  and  DIS.  There  are  two 
components.  The  first  is  the  development  of  the  Individual  Soldier  Mobility  Simulator  (ISMS), 
which  is  developing  the  capability  to  simulate  lower  body  movement.  The  second  is  developing 
an  improved  capability  for  TTES  to  track  weapon  position  and  upper  body  position  accurately. 

Two  less  tasks  have  been  excluded  from  this  list.  The  first.  Task  7,  Individual  Combatant 
Hardware  Suite,  is  a  hardware  and  software  acquisition.  The  second.  Task  8,  is  the  assessment 
described  in  this  report.  Neither  of  these  tasks  is  currently  funded. 
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Time  Constraints 


The  following  is  a  notional  timeline  based  on  these  assumptions:  all  assessment  activities 
must  be  completed  prior  to  30  September  1996;  some  form  of  contract  support  will  be  required 
for  the  conduct  of  most,  if  not  all,  task  assessments;  and  the  assessment  will  not  be  a  one-time 
activity,  but  a  series  of  events  occurring  over  a  period  of  time.  The  dates  below  describe  the 
general  time  frame  in  which  the  assessments  must  be  conducted. 

Begin  materials  preparation  (scenarios,  environments,  data  collection  instruments,  etc.)  -- 
Oct  95 

Begin  data  collection  -  Jan  96 
Begin  documentation  --  Jul  96 
Complete  all  tasks  --  Sep  96 


Performance  Method 

The  work  will  be  performed  as  a  series  of  tasks  which  can  be  carried  out  by  either  in-house 
government  organizations  (e.g.,  NAWC-TSD,  ARL-HRED),  the  task  contractor  (e.g.,  the 
Institute  for  Simulation  and  Training,  the  University  of  Peimsylvania,  etc.),  an  independent 
contractor,  or  a  combination  of  the  above.  Each  of  these  approaches  has  its  strengths  and 
weaknesses.  The  developing  organization  (government  or  contractor)  is  the  most  knowledgeable 
about  the  technology  involved,  but  may  not  have  the  manpower  or  skills  to  develop  the  necessary 
materials  and  conduct  the  assessment.  In  addition,  this  would  leave  the  integration  of  the 
individual  task  assessments  undone.  It  is  likely  that  no  single  independent  contractor  would  have 
the  required  skills  to  conduct  the  entire  assessment.  A  combination  of  developing  organization 
and  independent  contractor  could  do  all  of  the  work,  but  would  be  the  most  difficult  to  manage 
and  control. 


Types  of  Assessment  Approaches 

Given  the  diversity  of  the  task  objectives  and  the  methods  used  to  achieve  them,  several 
different  approaches  will  be  required.  This  section  describes  the  approaches  in  general  and  then 
identifies  which  approaches  are  feasible  for  each  task. 

Lessons  Learned 

It  is  anticipated  that  one  of  the  most  useful  types  of  results  that  can  be  obtained  from  an  ICSS 
will  be  the  "lessons  learned"  from  the  development  process.  This  includes  not  only  what 
worked,  but  what  did  not.  For  example,  in  the  area  of  control  of  direction  of  movement,  NAWC- 
TSD  has  rejected  an  approach  that  involved  a  pressure-sensitive  ring  on  the  floor,  and  ARI  has 
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rejected  an  approach  that  involved  tracking  shoulder  position.  It  is  important  to  document  why 
these  approaches  were  terminated.  The  assessment  of  each  task  will  include  lessons  learned.  It  is 
expected  that  lessons  learned  will  be  documented  by  the  responsible  government  organization 
and/or  developing  contractor,  with  input  from  and  coordination  with  the  other  activities  involved 
in  the  performance  of  that  task,  whether  government  or  contractor. 

Technical/User/Subiect  Matter  Expert  tSME'>  Review 

Technical  experts  can  review  and  assess  reports,  plans,  and  technical  documentation  to  assess 
their  soundness  and  quality.  Representative  users  (soldiers  and  trainers)  and  SMEs  can  provide  a 
variety  of  information  about  a  system  or  component  in  structured  situations  without  the  necessity 
of  conducting  a  full-scale  human  in  the  loop  experiment.  Issues  that  can  be  assessed  in  this 
manner  include  the  realism  of  a  simulation  or  human  figure,  the  similarity  of  different 
environments  or  objects,  the  appropriateness  of  various  tactical  or  maintenance  behaviors,  and 
user  acceptability.  The  important  information  that  comes  from  user  acceptability  is  an 
identification  of  the  strengths  and  apparent  weaknesses  of  a  system.  While  user  acceptance  does 
not  guarantee  that  a  system  will  train  successfully,  a  lack  of  user  acceptance  usually  constitutes  a 
major  obstacle  to  successful  implementation  and  utilization  in  the  field. 

Human  in  the  loop  evaluations 

A  critical  aspect  of  the  assessment  will  be  the  human  in  the  loop  evaluations.  Four  types  of 
soldier  in  the  loop  evaluations  are  possible. 

Real  world  vs  virtual  world  performance.  If  ICSS  technology  is  to  be  used  to  evaluate  new 
doctrinal  or  equipment  concepts  (for  example,  providing  each  squad  member  with  a  radio, 
improved  load-bearing  equipment,  or  an  improved  anti-personnel  weapon),  then  it  needs  to  be 
shown  that  performance  in  the  simulation  is  similar  to  performance  in  the  real  world,  not  for  all 
aspects  of  the  simulation,  but  for  those  derived  from  the  requirements.  For  example,  if  accurate 
simulation  of  movement  rates  is  required,  a  squad  should  not  move  tactically  at  radically 
different  rates  in  the  simulation  and  in  the  real  world.  If  accurate  simulation  of  small  arms  fire  is 
important,  then  an  individual's  marksmanship  performance  in  the  simulation  should  be  closely 
related  to  their  marksmanship  performance  in  the  real  world.  If  accurate  simulation  of 
maintenance  performance  is  important,  then  component  part  accessibility  and  task  completion 
times  should  be  close  to  those  found  in  the  real  world. 

Training  effectiveness  and  transfer.  If  ICSS  technology  is  to  be  used  to  train  soldiers  or 
marines  to  perform  combat  tasks,  then  it  needs  to  be  shown  that  performance  improves  with 
practice  in  the  simulation,  and  that  practice  in  the  simulation  improves  performance  of  the  task  in 
the  real  world.  As  an  example,  the  voice  and  gesture  recognition  system  could  be  used  to  train  a 
soldier  to  control  a  fire  team  moving  tactically.  A  group  of  trainees  could  be  put  through  a  short 
training  sequence,  perhaps  consisting  of  a  series  of  scenarios,  with  SME  trainers  providing 
feedback.  Improvement  could  be  assessed,  using  ratings  by  SMEs  or  objective  performance 
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measures.  Pretests  and  post-tests  consisting  of  field  exercises  could  be  used  to  assess  transfer  to 
the  real  world.  This  assessment  is  probably  the  most  difficult  and  resource-consuming  (time  and 
money)  to  conduct. 

Critical  interface  issues.  It  is  anticipated  that  discussions  among  task  leaders  will  identify 
soldier  interface  questions  that  can  be  resolved  by  human  in  the  loop  evaluations.  These  may 
include  such  things  as  the  advantages  of  various  forms  of  movement  control,  display  devices,  etc. 
Evaluations  can  be  performed  in  conjunction  with  the  performance  and  training  effectiveness 
evaluations,  or  as  stand-alone  experiments. 

Side  effects  and  after  effects.  Simulator  sickness  has  been  identified  as  a  potential  problem 
with  the  use  of  Virtual  Environments  for  training.  Generally  accepted  and  easily  administered 
methods  for  measuring  its  severity  exist.  Although  it  is  not  anticipated  that  any  experiments 
would  be  conducted  solely  for  the  purpose  of  collecting  data  on  simulator  sickness,  simulator 
sickness  data  should  be  collected  as  part  of  all  human  in  the  loop  experiments.  The  Simulator 
Sickness  Questionnaire  (SSQ;  Kennedy,  Lane,  Berbaum,  &  Lilienthal,  1993)  provides  a 
generally  accepted  means  of  assessing  simulator  sickness. 

The  purpose  of  the  human  in  the  loop  evaluations  is  to  obtain  data  to  support  future  design 
and  development.  They  are  not  intended  to  serve  as  “Go/No  Go”  tests. 

Assessment  Approaches  Matched  to  Tasks 

Table  2  shows  the  assessment  approaches  proposed  for  each  task.  A  detailed  discussion  of  the 
proposed  assessments  for  each  tasks  will  follow  in  a  later  section. 

Approach  to  the  Development  of  Lessons  Learned 

The  documentation  of  lessons  learned  will  be  a  part  of  the  task  assessment  for  every  task. 
These  lessons  learned  will  normally  be  developed  by  the  government  task  leader  and  the 
developing  contractor.  The  following  types  of  lessons  learned  are  to  be  included: 

Technical  approaches  that  did  not  work,  and  if  known,  the  reasons  why.  This  should 

include  hardware,  software,  networking,  and  human-computer  interface. 

Lfndocumented  characteristics  of  off-the-shelf  hardware  and  software. 

Successfiil  short  cuts. 

Lessons  learned  about  using  VR  for  training,  human  performance  measurement,  and 

experimentation. 

Management  and  scheduling  of  VR  development. 
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Table  2 


Assessment  Approaches  for  Each  Task 


Task 

Assessment  Approach 

Lessons 

Learned 

Human  in  the 
Loop 

Technical/ 

User/SME 

Review 

1 A  -  Jack  Combat 

Y 

Y 

N 

IB-  Jack  Maintenance 

Y 

N 

Y 

2  -  Voice/Gesture 

Y 

Y 

Y 

4  -  CCHC  (Includes  Task  3) 

Y 

Y 

Y 

5  -  Human  Figures/DIS 

Y 

N 

Y 

6A  -  ISMS 

Y 

Y 

N 

6B  -  Upper  Body  Tracking 

Y 

Y 

N 

Y  =  Yes  N  =  No 


The  sequence  of  activities  to  develop  the  lessons  learned  will  be  as  follows. 

Government  task  leader  and  developing  contractor  develop  draft  lessons  learned  for  their 
task(s). 

Government  task  leader  and  developing  contractor  present  their  lessons  learned  at  a 
contractor-run  conference  held  specifically  for  the  purpose  of  disseminating  these  lessons 
learned  within  the  ICSS  community  and  identifying  those  general  lessons  learned  that 
apply  across  multiple  tasks. 

Government  task  leader  and  developing  contractor  develop  a  written  report  chapter 
describing  their  lessons  learned. 

Contractor  organizes  and  integrates  chapters,  and  adds  an  introduction  and  conclusions,  to 
produce  a  final  lessons  learned  report. 
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Individual  Task  Assessment  Plans 


Task  1  A.  PIS  Control  of  Articulated  Human  Figure  Behaviors:  Combat 
Task  Description 

This  task  will  develop  and  demonstrate  the  capability  of  using  DIS  to  drive  graphic 
representations  of  human  figures  through  Jack,  a  software  tool  for  manipulating  and  displaying 
articulated  three-dimensional  human  forms. 

The  use  of  a  distributed  network  involves  multiple  individuals  interacting  with  each  other 
while  not  co-located.  The  DIS  communication  protocols  are  designed  to  provide  sufficient 
information  for  one  DIS  node  to  graphically  represent  the  behaviors  of  a  human  figure  (whether 
controlled  by  an  actual  human  or  computer  software)  at  another  DIS  node.  The 
following  behaviors  and  states  for  human  figures  are  being  addressed:  (1)  stance  (upright, 
kneeling,  or  prone),  (2)  movement  (heading  and  velocity),  (3)  impact  of  injury  or  dead ,  (4) 
status  of  weapons  (stowed  and  deployed),  (5)  identity  (fnendly,  hostile,  and  neutral),  (6)  use  of 
articulated  objects  (doors  and  windows),  (7)  use  of  movable  objects  (rocks,  chairs),  (8)  use  of 
hand  tools  and  (9)  use  of  common  support  equipment  (for  modeling  ground  maintenance 
activity). 

In  behaviors  1-4,  Jack'mW  provide  for  the  transition  between  DIS  posture  conditions.  For 
example,  when  the  "upright"  state  changes  to  a  "kneeling"  state.  Jack  will  provide  a  smooth 
transition  of  the  human  figure  between  these  two  postures.  In  behavior  5,  Jack  will  provide 
various  types  of  clothing  to  represent  fnendlies,  hostile  combatants,  and  neutrals.  DIS  does  not 
yet  have  precisely  defined  methods  for  communicating  behaviors  6-9,  and  Jack  does  not  have 
the  ability  to  portray  them  in  fine  detail.  Extensions  to  DIS  and  Jack  will  be  built  to 
accommodate  the  need  to  model  a  wider  range  of  human  behaviors  during  the  second  year  of 
work  on  this  task.  These  extensions  will  be  embedded  in  new/modified  network  communication 
protocols  that  permit  efficient  control  of  human  behaviors  over  DIS. 

The  approach  that  Jack  development  has  taken  to  satisfy  behaviors  1  through  4  while 
maintaining  an  acceptable  rate  of  speed  of  actions  is  a  combination  of  a  motion  library  and  level 
of  detail  modeling.  The  motion  library  is  based  on  the  fact  that  the  DIS  entity  state  protocol  data 
unit  (PDU)  can  recognize  only  a  limited  number  of  static  postures:  standing,  kneeling,  prone,  or 
dead.  For  the  first  three  of  these,  the  weapon  can  be  either  stowed  or  deployed.  The  University 
of  Pennsylvania  has  developed  posture  graphs  (see  Figure  1,  for  example)  which  show  the 
possible  transitions  among  the  available  postures.  For  simplification,  direct  transitions  among  all 
possible  postures  are  not  possible.  For  example,  a  crawling  figure  which  was  “killed”  would 
transition  first  to  a  prone  deployed  posture,  then  to  a  dead  posture,  rather  than  going  directly 
from  crawling  to  dead.  Three  animation  sequences,  using  different  level  of  detail  models,  are 
then  developed  for  each  of  these  possible  transitions.  The  level  of  detail  models,  called  human- 1, 
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human-2,  and  human-3,  differ  in  terms  of  a  variety  of  characteristics  (see  Table  3).  The  choice  of 
level  of  detail  models  is  dependent  on  the  distance  of  the  observer  to  the  figure. 


Figure  1.  The  static  posture  graph  (from  Granieri,  Crabtree,  &  Badler,  1995). 

As  described  by  Reece  (1994),  and  shown  in  Figure  2,  there  are  five  different  representations 
of  the  trainee's  human  figure  in  DIS:  (1)  the  true  position;  (2)  the  position  as  measured  by 
sensors;  (3)  the  position  as  reported  in  the  DIS  PDUs  transmitted  over  the  network;  (4)  the 
position  as  interpolated  by  the  receiving  simulator  platform;  and  (5)  the  position  as  rendered  on 
the  display  of  the  receiving  platform.  Each  transformation  introduces  possibilities  for  position 
and  temporal  errors.  This  could  be  particularly  important  for  combat  simulations,  where  the 
errors  could  well  affect  whether  or  not  a  human  figure  is  visible,  or  even  hit  or  killed. 

Assessment  Issues 

Are  the  actions  and  appearance  of  DIS-driven  human  figures  discemibly  different  when  their 
actions  are  driven  by  an  instrumented  human  and  when  their  actions  are  driven  by  computer 
models?  By  discemibly  different  we  mean:  can  human  observers  detect  a  difference  (at  different 
distances);  does  the  speed  with  which  selected  movements  are  made  differ? 
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Do  the  different  level  of  detail  Jack  models  differ  in  terms  of  their  likelihood  of  introducing 
location  errors? 

What  are  the  appropriate  ranges  for  the  use  of  each  of  the  level  of  detail  models? 


Table  3 

The  Different  Levels  for  Detail  for  the  Human  Figure  Models 


Human- 1 

Human-2 

Human-3 

Polygons 

2400 

500 

120 

Rigid  Segments 

69 

19 

12 

Joints 

73 

17 

11 

DOFs 

134 

50 

21 

Motion  (Hz) 

60 

30 

15 

Taken  from  Granieri,  Crabtree,  and  Badler  (1995) 


Figure  2.  Five  forms  of  entity  state  representation  in  a  DIS  system  (from  Reece,  1994). 
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General  Assessment  Approach 


A  human  in  the  loop  experiment  (I.A.)  will  be  conducted  to  address  these  issues.  A  series  of 
short  segments  from  a  combat  scenario  will  be  prepared.  Each  segment  will  consist  of  a  human 
figure  performing  an  action.  The  action  will  be  performed  by  either  JacA:  or  an  instrumented 
human.  Recordings  of  each  of  these  actions  will  be  converted  to  DIS  PDUs  and  the  PDUs  will 
be  used  to  drive  each  of  the  three  Jack  level  of  detail  models.  The  actions  of  the  “live”  humans 
will  also  be  videotaped  for  later  comparisons.  Human  observers  will  be  asked  to  determine 
whether  each  recording  shows  control  of  the  figure  by  a  human  or  a  computer.  Discrepancies  in 
the  positions  of  the  key  body  parts  (such  as  head,  hands,  and  feet)  between  the  originating  and 
receiving  platform  will  be  compared.  The  occurrence  of  such  "impossible"  events  as  floating  in 
the  air  or  sinking  into  the  groimd  will  be  determined  by  technical  experts. 

A  very  large  number  of  different  observations  are  possible.  Human  observers  could  view  each 
single  action  vmder  48  different  conditions.  Each  action  can  be  performed  by  two  types  of  figure 
control  (human  or  computer),  three  level  of  detail  models,  at  three  different  distances  and  from 
three  different  orientations  relative  to  the  figure  (front,  side,  or  rear).  The  distances  will  be 
selected  based  on  preliminary  experimentation,  but  are  expected  to  be  between  25  and  150 
meters. 

Four  different  types  of  human  figure  activities  can  be  examined  with  this  approach: 

Posture  Transitions.  Each  scenario  segment  will  consist  of  a  human  figure  performing  a 
simple  posture  transition  (either  standing  to  kneeling,  standing  to  prone,  kneeling  to  standing, 
kneeling  to  prone,  prone  to  standing,  or  prone  to  kneeling).  Human  observers  will  view  and 
categorize  as  human-or  computer-controlled  288  segments  (48  times  for  each  transition). 

Movement.  Each  scenario  segment  vvdll  consist  of  a  human  figure  either  running,  walking 
forward,  walking  backward,  or  crawling.  Human  observers  will  view  and  categorize  as  human- 
or  computer-controlled  192  segments  (48  times  for  each  type  of  movement). 

Obstacle  Avoidance.  Each  scenario  segment  will  consist  of  a  walking  human  figure  changing 
direction  to  avoid  an  obstacle,  then  returning  to  the  original  direction  of  movement.  Human 
observers  will  view  and  categorize  as  human-  or  computer-controlled  48  segments. 

Weapons  Firing.  Each  scenario  segment  will  consist  of  a  human  figure  firing  a  weapon  (Ml  6) 
from  either  a  standing,  kneeling,  or  prone  position.  Human  observers  will  view  and  categorize  as 
human-  or  computer-controlled  144  segments. 

One  of  the  critical  activities  in  preparing  for  this  experiment  will  be  to  reduce  the  number  of 
possible  observations  to  a  reasonable  number  (approximately  100)  while  still  providing  an 
adequate  sampling  of  distances,  activities,  positions,  and  orientations. 
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Scenario  and  terrain  database 


Short  combat  segments  in  the  Quantico  Combat  Training  Village  or  the  ARI  open  terrain 
database. 

Performance  Measures 

Ratings  of  Figure  Similarity 
Simulated  Position  Discrepancies 
Polygons  generated  per  unit  time 

Assessment  Tasks:  Experiment  1  A. 

Task  lA.l.  Develop  Scenario  or  Scenario  Segments.  From  the  672  different  scenario 
segments  described  above,  select  approximately  100  for  inclusion  in  the  experiment.  Develop 
both  Jac^-controlled  and  human-controlled  versions  of  each  segment. 

Task  1A.2.  Develop  and  test  Experimental  Control  and  Data  Collection  Software.  This 
software  will  control  the  presentation  of  the  video  segments,  time  and  record  the  subject's 
responses  (if  entered  directly  into  the  computer),  and  create  a  datafile  for  further  analysis. 

Task  1A.3.  Develop  Experimental  Procedures  and  Materials  (data  collection  forms, 
instructions  for  Subjects,  etc.).  This  should  include  the  development  of  forms  for  collecting 
backgroxmd  information,  vision  testing,  and  preparation  of  the  necessary  documents  for  approval 
of  the  use  for  human  subjects. 

Task  1A.4.  Conduct  Experiment. 

Task  1A.5.  Analyze  Data. 

Task  lA.5.a.  Analyze  Human  Performance  Data  (accuracy  and  speed  of  human 
subject  responses). 

Task  lA.5.b.  Analyze  Hardware/Software  Performance  Data  (position 
discrepancies  between  send  and  receive  nodes). 

Task  1A.6.  Write  Report. 
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Task  IB.  PIS  Control  of  Articulated  Human  Fi^e  Behaviors:  Maintenance 


Task  Description 

This  task  shares  many  aspects  of  task  1  A,  except  that  the  focus  is  on  the  use  of  human  figure 
models  (specifically  Jack)  for  maintenance,  rather  than  combat  tasks.  As  noted  previously,  the 
more  challenging  modifications  and  extensions  of  DIS  and  Jack  are  needed  more  for  this 
application  than  for  the  combat  scenario.  Important  behaviors  are:  use  of  articulated  objects 
(doors  and  windows),  movable  objects  (rocks,  chairs),  hand  tools,  and  common  support 
equipment.  The  intended  use  of  the  human  figure  model  is  also  different.  The  intended  me  of 
Jack  in  maintenance  applications  is  to  support  the  design  of  new  maintenance  systems  and 
procedures.  Thus  whether  the  human  figure  "looks  like"  an  actual  human  is  relatively 
unimportant.  The  crux  of  the  issue  is  whether  the  use  of  Jack,  in  conjunction  with  a  model  of  the 
system  or  equipment  to  be  evaluated  or  tested,  can  predict  human  performance  accurately  enough 
to  lead  to  appropriate  design  decisions. 

Assessment  Issues 

Is  the  behavior  of  actual  humans  and  Jack  models  (when  driven  by  appropriate  task  models) 
sufficiently  similar  to  result  in  the  same  system  design  decisions  or  maintenance  procedures? 
Specifically,  are  task  completion  times  similar,  are  the  sequences  of  actions  taken  similar,  and 
are  the  same  actions  possible  (i.e.,  accessibility)? 

General  Assessment  Approach 

This  assessment  needs  to  be  done  with  an  existing  system,  not  a  conceptual  system,  so  that 
actual  human  performance  data  can  be  collected  and  compared  with  Jack  performance  data.  It 
may  not  be  necessary  to  use  a  complete  system,  but  there  should  be  a  reasonable  sampling  of  the 
types  of  tasks  on  which  Jack  is  expected  to  be  used.  It  may  also  be  possible  to  use  high  fidelity 
mock-ups  or  simulators  in  place  of  the  actual  equipment  in  order  to  obtain  human  performance 
data.  Either  approach,  or  a  combination  of  both,  should  suffice.  In  either  case,  the  first  step  is  to 
measure  the  performance  of  several  different  humans  on  the  task.  Next  measure  Jack 
performance  on  the  same  tasks.  Use  maintenance  SMEs  to  judge  the  similarity  of  Jack  and 
human  performance.  In  addition,  make  objective  comparisons  of  task  completion  times  and 
sequences  of  actions.  Unlike  task  1  A,  in  Task  IB  similarity  refers  to  the  sequence  and  timing  of 
maintenance  actions,  not  to  similarity  of  appearance.  The  assessment  needs  to  focus  on  those 
new  capabilities  that  are  being  added  to  Jack  as  part  of  the  ICSS  effort. 

A  potential  problem  with  this  approach  is  that  no  standards  currently  exist  for  how  similar 
human  performance  and  Jack  performance  need  to  be.  With  use  of  multiple  humans,  and 
multiple  runs  with  Jack,  performance  distributions  could  be  obtained  for  both,  and  statistical  tests 
run,  but  sufficient  runs  to  make  these  tests  meaningful  may  not  be  possible.  Possibly,  objective 
standards  could  be  derived  from  SMEs  or  constructive  models.  Because  of  the  complexity  of 
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this  assessment,  a  three-phase  approach  (initial  small-scale  assessment,  revision  of  models  and 
assessment  procedures,  and  final  large-scale  assessment)  is  recommended. 

Scenario  Terrain  Database 


Maintenance  performance  at  an  air  base. 

Performance  Measures 

SME  Ratings  of  similarity  of  performance 
Task  action  sequences 
Task  times 

Assessment  Tasks 

Task  IB.l.  Develop  maintenance  scenario  or  scenario  segments.  Subtasks  will  include:  the 
selection  of  the  system  or  the  tasks  that  are  to  be  the  focus  of  the  assessment;  the  development  of 
the  task  models  to  drive  Jack,  and  obtaining  or  developing  the  equipment,  simulators,  or  mock- 
ups  that  will  be  used  by  the  human  maintainers. 

Task  1B.2.  Execute  selected  maintenance  scenarios  using  Jack.  Record  data  on  task 
sequences  and  task  completion  times. 

Task  1B.3.  Execute  selected  maintenance  scenarios  using  actual  maintenance  technicians. 
Record  data  on  task  sequences  and  task  completion  times. 

Task  1B.4.  Develop  experimental  procedures  and  materials  (procedure  similarity  rating 
schemes,  task  standards,  data  collection  forms,  instructions  for  SMEs,  etc.). 

Task  1B.5.  Use  SMEs  to  judge  similarity  of  Jack  and  human  maintenance  technician 
procedures. 

Task  1B.6.  Analyze  Data. 

Task  lB.6.a.  Analyze  SME  Rating  Data 
Task  lB.6.b.  Analyze  Objective  Performance  Data 
Task  1B.7.  Write  Report. 
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Task  2.  Linking  Individual  Combatants  with  Computer  Controlled  Representations  of 

Squad/Fire  Team  Members 


Task  Description 

The  purpose  of  this  effort  is  to  develop  a  prototype  of  a  more  realistic  interface  for  the 
dismounted  squad  or  foe  team  leader  in  the  DIS  battlefield  environment.  The  emphasis  in  DIS  to 
date  has  been  on  the  simulation  of  combat  for  soldiers  fighting  from  vehicles,  not  for  dismounted 
soldiers.  However,  the  cluster  of  technologies  generally  referred  to  as  Virtual  Environment  (VE) 
technology  has  the  capability  to  integrate  the  dismounted  soldier  more  fully  into  DIS.  This  effort 
will  produce  a  prototype  system  using  voice  recognition  and  limb  tracking  technologies  to  enable 
a  squad  or  fire  team  leader  to  control  dismounted  Computer-Generated  Forces  (CGF)  combatants 
(representing  squad  or  fire  team  members)  through  a  set  of  verbal  commands  and  hand  and  arm 
signals.  The  commands  implemented  are  shown  in  Figure  3.  This  approach  will  permit  training 
the  small  unit  leader  in  those  decision-making,  communication,  and  leadership  skills  in  a  realistic 
environment  with  the  necessity  of  equipping  an  entire  squad  of  soldiers. 


Helicopter  Gesture  Commands 

Squad  Voice  Commands 

Squad  Gesture  Commands 

Prepare  for  Guidance 

Fire/Move/Rush 

Form  Column 

Forward 

There 

Form  Line 

Backward 

NowA/Vhen  he 

Form  Wedge 

Move  Right 

Fires/Moves/Rushes 

Close  Up 

Move  Left 

Open  Up 

Upward 

Disperse 

Downward 

Increase  Speed 

Land 

Decrease  Speed 

Change  Direction 

Point  to 

Look  at 

Commence  Firing 

Cease  Firing 

Figiue  3.  Voice  and  gesture  commands. 
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Commercial  off-the-shelf  hardware  and  software  is  being  used  for  the  voice  recognition 
subsystem.  The  gesture  recognition  subsystem  requires  the  development  of  completely  new 
software  and  modifications  to  existing  hardware.  Position  of  the  arms  (relative  to  the  torso)  is 
tracked  using  three  electromagnetic  trackers.  Specially  designed  gloves  are  used  to  determine 
whether  the  hands  are  open  or  closed.  The  software  recognizes  both  dynamic  and  static  gestures, 
and  is  expandable  in  terms  of  the  number  of  gestures  that  can  be  recognized.  The  Computer- 
Generated  Forces  Testbed  (CGFT)  is  being  modified  to  receive  the  voice  and  gesture  commands. 
A  DIS  interface  is  being  developed  so  that  the  soldier  interface  can  communicate  with  CGFT. 

Assessment  Issues 

How  accurately  are  commands  (both  voice  and  gesture)  recognized  as  a  function  of  individual 
characteristics  (body  size,  practice,  skill  or  experience  in  using  gestures  for  command  and 
control)? 

Does  skill  in  giving  gesture  commands  improve  with  practice? 

Does  tactical  proficiency  improve  with  practice? 

Does  the  simulated  fiiendly  force,  at  the  unit  level,  respond  appropriately  given  the  tactical 
situation  and  the  commands  given? 

General  Assessment  Approach 

Two  human  in  the  loop  experiments  will  be  conducted.  The  first  (Experiment  2.1)  will  address 
the  first  two  issues.  The  second  (Experiment  2.2)  will  address  the  last  two  issues.  In  the  first 
experiment,  an  instrumented  human  (the  trainee)  will  serve  as  a  squad  or  fire  team  leader  in  a 
combination  of  short  combat  scenarios.  Scenarios  could  be  Military  Operations  in  Urban 
Terrain/Military  Operations  in  Built-up  Areas  (MOUT/MOBA)  maneuvering  and  engaging 
targets  but  should  also  involve  assumption  of  various  movement  formations.  The  leader  will  be 
videotaped  and  his  voice  commands  recorded  during  the  conduct  of  the  scenario.  Recordings 
will  be  made  of  each  gesture/command  as  interpreted  by  the  computer,  and  these  will  be 
compared  with  the  interpretations  of  human  judges  based  on  the  videotapes.  Using  these  data,  it 
will  be  possible  to  look  at  voice  and  gesture  recognition  performance  as  a  function  of  a  number 
of  variables.  The  initial  experiment  will  compare  inexperienced  (college  students)  with 
experienced  (military  personnel)  leaders  and  will  also  examine  the  effects  of  practice  with  the 
system  on  performance. 

In  the  second  experiment,  the  system  will  be  used  to  train  tactics  and  command  and  control  to 
Army  Reserve  Officers  Training  Corps  (ROTC)  students  or  other  leader  trainees.  This  would 
use  the  same  tactical  scenarios  as  the  first  experiment,  but  would  look  at  the  changes  in  tactical 
proficiency  over  time  as  assessed  by  SMEs.  At  the  same  time,  the  SMEs  could  assess  the 
appropriateness  of  the  simulated  friendly  forces  given  the  commands  and  the  tactical  situation. 
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Scenario  and  Terrain  Database 


Combat  scenario  using  the  ARI  Rural  Terrain  or  Quantico  Combat  Training  Village  database. 


Performance  Measures 


Gesture  Recognition  Accuracy 

Tactical  Proficiency  (SME  Ratings) 

Appropriateness  of  CGF  behaviors  (SME  Ratings) 

Assessment  T asks :  Experiment  2 . 1 

Task  2.1.1  ■  Develop  Scenarios.  These  will  be  a  series  of  relatively  short  combat  or 
movement  scenarios. 

Task  2.1.2.  Develop  and  test  Experimental  Control  and  Data  Collection  Software.  This 
software  will  control  the  presentation  of  the  scenarios,  and  record  the  subject's  limb  positions  and 
how  they  were  interpreted  as  gestures  in  a  datafile  for  further  analysis. 

Task  2.1.3.  Develop  Experimental  Procedures  and  Materials  (data  collection  forms, 
instructions  for  Subjects,  instructions  forjudges,  etc.).  This  should  include  the  development  of 
forms  for  collecting  background  information,  vision  testing,  and  preparation  of  the  necessary 
documents  for  approval  of  the  use  for  human  subjects. 

Task  2.1.4.  Conduct  Experiment. 

Task  2.1 .5.  Analyze  Data. 

Task  2. 1.6.  Write  Report. 

Assessment  Tasks:  Experiment  2.2 

Task  2.2.I.  Develop  Scenarios  It  should  be  possible  to  use  the  same  scenarios  as  were  used 
in  Experiment  2.1.  However,  some  modifications  may  be  required. 

Task  2.2.2.  Develop  and  test  Experimental  Control  and  Data  Collection  Software.  Again,  the 
control  software  used  in  Experiment  2.1  should  suffice,  with  minor  modifications. 

Task  2.2.3.  Develop  Experimental  Procedures  and  Materials  (data  collection  forms, 
instructions  for  Subjects,  etc.)  This  should  include  the  development  of  forms  for  collecting 
background  information,  vision  testing,  and  preparation  of  the  necessary  documents  for  approval 
of  the  use  for  human  subjects. 
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Task  2.2.4.  Conduct  Experiment. 
Task  2.2.5.  Analyze  Data. 

Task  2.2.6.  Write  Report. 


Task  4.  PIS  Computer  Controlled  Hostile  Combatants  ( CCHO 

(Note:  This  plan  assrimes  that  products  from  TASK  3,  SELECTED  ACTIONS  OF 
INDIVIDUAL  NEUTRALS  AND  HOSTILE  COMBATANTS,  will  have  been  incorporated  into 
Task  4,  and  that  no  separate  assessment  of  Task  3  products  will  be  performed.) 

Task  Description 

This  task  will  identify  and  define  techniques,  tactics  and  behaviors  employed  by  hostile 
combatants  and  neutrals  for  virtual  environments  and  DIS.  The  types  of  combatants  which  the 
infanhyman  will  encounter,  and  the  actions  of  these  various  types  of  combatants  will  be 
considered.  Hostile  combatants  range  from  the  untrained  civilian  carrying  a  weapon,  to  the 
soldier  who  has  received  general  training,  to  the  highly  trained  warrior.  Hostile  actions  can  be 
broken  down  into  basic  behaviors,  techniques,  and  tactics.  Basic  behaviors  are  physical 
movements  such  as  running,  carrying  a  weapon,  and  crouching.  Techniques  are  those  behaviors 
used  in  a  tactical  manner,  such  as  running  while  maintaining  cover,  carrying  a  weapon  in  a 
manner  that  allows  rapid  use  of  the  weapon,  and  crouching  behind  structures  that  offer 
concealment  and  cover.  Tactics  are  coordinated  group  actions  made  to  achieve  tactical  goals. 

Computer  controlled  representations  of  hostile  combatants  will  be  developed  and 
implemented  to  interact  with  the  dismounted  DIS  trainee.  The  Institute  for  Simulation  and 
Training  (1ST)  CGFT  will  serve  as  the  basis  of  this  effort.  The  CGFT  currently  models  each 
individual  of  a  rifle  squad  that  engages  armored  vehicles.  These  CCHCs  will  be  adapted  to 
engage  individual  combatants  rather  than  vehicles. 

The  general  architecture  provides  for  a  DIS-compatible  distributed  network  consisting  of 
multiple  TTES  stations  and  a  single  CCHC  station.  Both  the  TTES  and  CCHC  nodes  contain  a 
database  of  the  geographical  region  in  which  the  exercises  occur  (i.e.,  Quantico  Combat  Training 
Village).  The  overall  technical  approach  is  to  modify  the  SAFOR  version  of  the  CGFT.  Areas 
that  will  require  extensive  modifications  to  meet  the  requirements  of  TTES  include  the  physical 
modeling  modules  (e.g.,  line  of  sight,  detection,  and  weapons  effects),  behavior  generation  (e.g., 
route  planner  for  inside  buildings,  specific  tactical  behaviors  occurring  in  urban  conflicts),  and 
the  capability  to  run  fully  autonomously. 
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Assessment  Issues 


There  are  two  classes  of  assessment  issues.  The  broad  assessment  issue  is  concerned  with  the 
contribution  of  the  CCHC  to  the  training  effectiveness  of  the  overall  system  of  which  it  is  a  part 
(in  this  case,  TTES).  The  specific  assessment  issues  are  concerned  with  the  fimctioning  and 
effectiveness  of  specific  CCHC  capabilities. 

The  broad  assessment  issue  is  whether  CCHC  enhances  the  practice  and  learning  of  tactically 
correct  behavior  by  marines  and  soldiers.  While  this  is  the  critical  CCHC  issue,  it  is  difficult  to 
assess  accurately.  The  CCHC  is  one  component,  and  only  one  component,  of  a  larger  training 
system.  The  visual  depiction  of  the  terrain  database,  the  characteristics  of  objects  in  the 
database,  and  the  representation  of  the  dynamic  processes  (e.g.,  structures  vs.  munitions)  in  the 
simulated  environment,  as  well  as  the  characteristics  of  the  trainee  interface  with  the  system  and 
the  instructional  strategies  employed  (e.g.,  AARs)  also  influence  training  effectiveness. 
Therefore,  a  comparison  of  TTES  with  some  other  treiining  approach  does  not  adequately  address 
the  contribution  of  CCHC  to  training  effectiveness.  A  comparison  of  two  versions  of  TTES, 
with  and  without  CCHC,  or  with  different  versions  of  CCHC,  is  required. 

Specific  assessment  issues  are: 

Can  the  environment  be  represented  in  a  manner  which  makes  it  possible  for  computer- 

controlled  entities  to  extract  tactical  information? 

Can  the  CCHC  drive  human  figure  models  through  DIS? 

Is  the  CCHC  behavior  tactically  sound?  (The  emphasis  should  be  more  on  how  well  it  fits 

common  sense  rules  of  behavior  than  tactical  sophistication.) 

Do  the  CCHC  perception  and  weapons  models  result  in  realistic  behavior? 

Is  the  performance  of  CCHC  interruptable,  believable,  and  indistinguishable  from  human- 

driven  icons? 

General  Assessment  Approach 

The  broad  assessment  issue  is  in  many  ways  more  suitable  for  empirical  assessment  than  the 
specific  ones.  The  general  approach  proposed  for  its  resolution  is  to  place  a  number  of  teams  or 
small  units  through  repetitions  of  a  tactical  (MOUT)  scenarios,  with  sufficient  variations  so  that 
the  scenario  cannot  be  learned,  and  assess  changes  in  performance  with  practice.  One-half  of  the 
groups  of  trainees  will  train  against  a  sophisticated  CCHC  (tactically  sotmd  behavior,  realistic 
perception  model,  and  realistic  weapons  effects).  The  other  half  will  train  against  a  rudimentary 
version  of  the  CCHC  (limited  tactical  capabilities,  simple  perception  model,  and  simple  weapons 
effects).  Both  groups  will  then  be  tested  in  an  instrumented  live  combat  scenario  against  a  live 
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opponent  in  the  Quantico  Combat  Training  Village  (using  MILES),  or,  as  a  less  desirable 
alternative,  a  TTES  scenario  against  the  sophisticated  CCHC.  Both  expert  ratings  and  objective 
measures  of  performance  will  be  obtained. 

Whether  or  not  the  environment  can  be  represented  in  a  manner  which  enables  the  CCHC  to 
extract  tactical  information,  and  whether  the  CCHC  can  drive  human  figure  models  through  DIS 
are  issues  more  suitable  for  assessment  as  lessons  learned,  rather  than  empirical  assessments. 

The  remainder  of  the  focused  issues  can  be  best  addressed  by  using  SMEs  to  view  the  behavior 
of  the  CCHC  in  highly  structured  scenarios  and  situations.  Examples  of  the  behaviors  that  might 
be  used  are: 

Immediately  move  to  the  nearest  covered  or  concealed  position  upon  receiving  fire; 

Plan  and  execute  a  route  to  a  designated  location; 

Plan  and  execute  a  covered  or  concealed  route  to  a  designated  location; 

From  a  defensive  position,  fire  upon  an  attacking  enemy  as  soon  as  he  is  detected; 

From  a  prone  position,  rise  and  "run"  into  a  nearby  doorway;  and 
Follow  another  squad  or  team  member. 

The  SMEs  will  rate  the  appropriateness  of  the  CCHC  tactical  behavior,  and  the  realism  of  the 
CCHC  behavior/movements.  For  comparison  purposes,  ratings  of  human-driven  icons 
performing  the  same  movements  will  also  be  obtained. 

Scenario  Terrain  Database 

Combat  (MOUT)  scenario  using  the  Quantico  Combat  Training  Village 
Performance  Measures 


SME  Ratings  of  Tactical  Performance 
Tactical  Mission  Accomplishment 
Survivability  or  Loss  Ratio 
Hits/Shots  &  Kills/Shots 
Engagement  Times  &  Ranges 

Assessment  Tasks:  Experiment  4. 1 

Task  4.1 .1.  Develop  Scenarios  and  Rudimentary  CCHC.  The  scenarios  should  be  multiple 
variations  on  a  common  training  objective  or  drill,  such  as  responding  to  contact  with  the  enemy 
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or  employing  overwatch  to  move  through  an  area.  The  rudimentary  CCHC  should  possess  less 
sophisticated  capabilities  than  the  CCHC  currently  available  at  the  time  of  the  evaluation.  For 
example,  the  rudimentary  CCHC  might  not  incorporate  target  motion  in  determining  whether  or 
not  a  target  is  detected,  or  might  not  include  own  firing  position  (e.g.,  standing,  prone)  in 
calculating  hit  probabilities. 

Task  4.1.2.  Develop  and  test  Experimental  Control  and  Data  Collection  Software.  This 
software  will  control  the  presentation  of  the  scenarios,  and  the  recording  of  the  participant's 
actions,  to  include  movements,  rounds  fired,  hits,  kills,  etc.,  and  create  a  datafile  for  further 
analysis. 

Task  4.1.3.  Develop  Experimental  Procedures  and  Materials  (data  collection  forms, 
instructions  for  Subjects,  etc.).  This  should  include  the  development  of  forms  for  collecting 
background  information,  vision  testing,  and  preparation  of  the  necessary  documents  for  approval 
of  the  use  for  human  subjects. 

Task  4.1.4.  Conduct  Experiment. 

Task  4.1.5.  Analyze  Data. 

Task  4.1.6.  Write  Report. 

Assessment  Approach:  Experiment  4.2 

Task  4.2.1.  Develop  Scenario  Segments.  This  will  include  the  selection  of  the  CCHC 
behaviors  to  be  addressed,  and  the  creation  of  each  of  those  scenarios  using  both  human 
controlled  and  computer-controlled  hostiles. 

Task  4.2.2.  Develop  and  test  Experimental  Control  and  Data  Collection  Software.  This 
software  will  control  the  presentation  of  the  video  segments,  time  and  record  the  subject's 
responses  (if  entered  directly  into  the  computer),  and  create  a  datafile  for  further  analysis. 

Task  4.2.3.  Develop  Experimental  Procedures  and  Materials  (data  collection  forms, 
instructions  for  Subjects,  etc.)  This  should  include  the  development  of  forms  for  collecting 
backgroimd  information,  vision  testing,  and  preparation  of  the  necessary  documents  for  approval 
of  the  use  for  human  subjects. 

Task  4.2.4.  Conduct  Experiment. 

Task  4.2.5.  Analyze  Data. 

Task  4.2.6.  Write  Report. 
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Task  5.  PIS  Cataloging  and  Auditing 


Task  Description 

This  task  will  analyze  the  adequacy  of  DIS  database  requirements  and  networking  protocols 
for  the  dismounted  combatant  in  virtual  environments.  DIS  has  established  standards  for 
networking,  protocols,  database  requirements,  and  environmental  factors.  They  have  not  been 
specifically  applied  to  dismoimted  combatants  operating  in  distributed  synthetic  environments. 
There  is  currently  no  systematic  examination  or  consensus  within  the  DIS  community  regarding 
the  adequacy  of  current  DIS  standards  to  support  representation  of  the  dismounted  combatant.  A 
systematic  examination  requires  detailed  analysis  and  evaluation  of  the  requirements  to  transmit 
information  about  the  behavior  of  dismounted  combatants  generated  by  the  work  conducted 
under  the  other  tasks;  comparison  of  the  requirements  with  the  capability  of  the  current  DIS 
protocol  structure;  and  the  development  and  promotion  of  solutions  which  resolve  the 
discrepancies. 

Assessment  Issue 

Are  the  proposed  approaches  for  modifying  DIS  standards  to  accommodate  human  figure 
requirements:  (a)  consistent  with  existing  or  proposed  DIS  standards;  and  (b)  adequate  to  satisfy 
other  non-ICSS  requirements  for  the  use  of  human  figures  in  DIS? 

General  Assessment  Approach 

The  draft  position  paper  resulting  fi-om  this  approach  will  be  reviewed  sequentially  by:  (a) 
task  leaders  and  their  performing  contractors;  (b)  the  members  of  the  DIS  Standards  Human 
Figures  Special  Interest  Group;  and  (c)  appropriate  DIS  standards  Working  Groups  after  it  has 
been  prepared  as  a  position  paper.  Prior  to  the  start  of  the  formal  assessment,  the  task  leaders 
and  their  performing  contractors  will  have  reviewed  and  commented  on  the  draft  position  paper, 
and  it  will  have  been  disseminated  electronically  to  members  of  the  DIS  Standards  Human 
Figures  Special  Interest  Group. 

Scenario  and  Terrain  Database 

N/A 

Performance  Measures 


N/A 
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Assessment  Tasks 


Task  5.1.  Present  position  paper  at  DIS  Standards  Conference  and  revise  based  on  feedback. 

Task  5.2.  Prepare  formal  position  paper  and  coordinate  it  with  the  DIS  Standards  Committee 
and  other  DIS  Standards  Working  Groups. 

Task  6A.  Human  Interface  for  DIS 


Task  Description 

An  essential  element  of  this  effort  is  the  design  and  fabrication  of  a  single  prototype  of  an 
Individual  Soldier  Mobility  Simulator  (ISMS).  Currently,  an  individual  soldier  interacts  with 
distributed  simulations  using  a  hand  driven  haptic  device  such  as  a  spaceball  or  joystick.  Neither 
these  devices  nor  powered  treadmills  allow  free,  natural  movement  of  the  combatant’s  hands  and 
feet,  synchronized  with  terrain,  in  a  virtual  environment.  To  address  this  deficiency,  a  lower 
torso  manipulator  worn  by  or  affixed  to  the  combatant  and  responsive  to  digital  commands 
generated  by  a  simulator  is  required.  The  device  will  supply  information  to  the  simulation 
indicating  the  combatant's  movements  through  the  virtual  terrain.  The  mobility  simulator  will 
allow  unrestricted  joint  movements  so  that  the  natural  feel  of  walking  or  running  on  terrain  is 
simulated  by  arresting  foot  movement  in  synchrony  with  foot  impacts  on  the  virtual  terrain.  The 
system  will  allow  the  combatant  to  move  naturally  in  all  directions  (three  degrees  of  translational 
freedom),  providing  six  degrees  of  freedom  (DOF)  total  (three  translational  DOF  per  foot)  and 
accommodate  the  5th  to  95th  percentile  soldier.  The  system  will  also  allow  the  combatant  the 
full  use  of  his  hands.  This  task  will  develop  a  cost-effective  mobility  simulator  that  can  be  used 
for  virtual  battlefield  training  by  the  Battle  Labs  and  DIS.  Two  mobility  simulators  have  been  or 
are  being  developed  as  part  of  this  effort.  The  first,  the  I-Port  "Quick  Fix"  system,  is  a  pedestal 
moimted,  pedal-driven  simulator.  The  second,  the  ISMS,  will  permit  a  more  natural  walking 
motion,  based  on  different  technology. 

Assessment  Issue 

How  does  energy  expenditure  using  the  I-Port  "Quick  Fix"  pedal  driven  system  and  the 
Individual  Soldier  Mobility  Simulator"  compare  with  energy  expenditure  while  performing  the 
same  tasks  in  the  real  world? 

General  Assessment  Approach 

A  series  of  human  in  the  loop  experiments  will  be  conducted  to: 
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Compare  energy  expenditure  while  walking  at  varying  speeds  and  inclines  using  the  I-Port 
Quick  Fix  system  (pedestal  mounted  and  pedal-driven)  with  that  predicted  by  empirically- 
based  models: 

Compare  energy  expenditure  and  movement  rates  using  the  I-Port  Quick  Fix  and  ISMS 
while  traversing  simulated  obstacle  courses  with  real  world  energy  expenditure. 

Scenarios  and  Terrain  Databases 

None  required  for  the  first  experiment.  Two  courses  (the  K-D  Range,  a  relatively  short 
course  with  many  obstacles,  and  a  longer  cross  country  course)  for  the  second. 

Performance  Measures 

Course  completion  time 
V02  (Oxygen  consumption) 

Assessment  Tasks:  Experiment  6A.1 

Task  6A.1.1.  Develop  Scenarios.  The  scenarios  will  require  the  subjects  to  “walk”  straight 
ahead  at  a  fixed  rate  of  speed  (2, 3,  or  4  miles  per  hour)  on  a  simulated  fixed  incline 
(-7.5,  -5.0,  -2.5, 0,  +2.5,  +5.0,  or  +7.5  degrees). 

Task  6A.1.2.  Develop  and  Test  Experimental  Control  and  Data  Collection  Software.  This  will 
require  that  the  incline  and  speed  sequences  be  programmed  into  the  I-Port  “Quick  Fix”  system, 
along  with  any  automated  data  collection  routines  required. 

Task  6A.1.3.  Develop  Experimental  Procedures  and  Materials  (data  collection  forms, 
instructions  for  Subjects,  etc.).  Equipment  to  monitor  heart  rate  and  V02  consumption  will  need 
to  be  obtained.  Forms  for  collecting  background  information,  and  the  necessary  documents  for 
approval  of  the  use  for  human  subjects  will  need  to  be  prepared.  Commercial  instruments  for 
measuring  heart  rate  and  V02  may  need  to  be  obtained. 

Task  6A.1.4.  Conduct  Experiment. 

Task  6A.1.5.  Analyze  Data. 

Task  6A.1.6.  Write  Report. 

Assessment  Tasks:  Experiment  6A.2 

Task  6A.2.1.  Develop  Terrain  Databases.  Two  terrain  databases  are  required:  the  K-D 
Range  and  the  obstacle  course.  Both  will  require  a  relatively  high  level  of  detail. 
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Task  6A.2.2.  Develop  and  test  Experimental  Control  and  Data  Collection  Software. 


Task  6A.2.3.  Develop  Experimental  Procedures  and  Materials  (data  collection  forms, 
instructions  for  Subjects,  etc.)  This  should  include  the  development  of  forms  for  collecting 
background  information,  and  preparation  of  the  necessary  documents  for  approval  of  the  use  for 
human  subjects. 

Task  6A.2.4.  Conduct  Experiment. 

Task  6A.2.5.  Analyze  Data. 

Task  6A.2.6.  Write  Report. 


Task  6B.  Upper  Body  Tracking 


Task  Description 

A  tracking  system  is  being  developed  which  will  be  able  to  monitor  the  position  of  a 
simulated  rifle  more  accurately.  Current  technology  provides  an  accuracy  comparable  to  a  59 
inch  dispersion  at  100  yards,  compared  to  a  requirement  for  4.6  inch  dispersion  and  100  yards, 
based  on  the  accuracy  of  the  Ml  6  rifle.  The  new  system  will  use  cameras  to  monitor  the 
positions  of  passive  markers  on  the  weapon. 

Assessment  Issues 


Does  improved  weapon  system  and  upper  body  tracking  accuracy  affect  the  accuracy  of 
simulated  weapons  firing  or  the  soldier's  perception  of  the  accuracy  of  the  simulated  firing? 

Does  the  improved  tracking  system  affect  soldier  comfort  or  side  effects? 

General  Assessment  Approach 

A  human  in  the  loop  experiment  will  be  conducted  to  determine  the  effects  of  the  improved 
tracking  system  (relative  to  a  baseline  system)  on  the  accuracy  of  firing  at  simulated  stationary 
and  moving  targets.  The  experiment  would  involve  two  groups  of  ten  subjects  each,  with  one 
group  using  the  existing  tracking  system  and  one  group  using  the  new  tracking  system.  Subjects 
will  be  stationary.  Targets  will  be  presented  at  four  different  distances  (to  be  determined  later) 
and  will  be  stationary  or  moving  at  a  constant  speed  in  one  of  eight  directions  relative  to  the 
subject  (vrith  the  subject  facing  12  o’clock,  the  target  would  be  moving  toward  either  12  o’clock, 
1:30, 3  o’clock,  4:30, 6  o’clock,  7:30, 9  o’clock,  or  10:30.  Subjects  will  fire  from  the  standing, 
kneeling,  and  prone  positions. 
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Scenarios  and  Terrain  Database 


A  target  engagement  scenario  (probably  a  modification  of  a  MOUT  scenario)  in  the  Quantico 
Combat  Training  Village. 

Performance  Measures 


Percent  hits 

Perceived  weapon  system  accmacy 
Comfort  and  convenience  questionnaires 
Simulator  Sickness  Questionnaire 

Assessment  Tasks 

Task  6B.1.  Develop  Target  Engagement  Scenario. 

Task  6B.2.  Develop  and  test  Experimental  Control  and  Data  Collection  Software.  This 
software  will  control  the  presentation  of  the  targets,  record  the  time  and  accuracy  of  the  subject's 
firing,  and  create  a  datafile  for  furiher  analysis. 

Task  6B.3.  Develop  Experimental  Procedures  and  Materials  (data  collection  forms, 
instructions  for  Subjects,  etc.).  This  should  include  the  development  of  forms  for  collecting 
background  information,  vision  testing,  and  preparation  of  the  necessary  documents  for  approval 
of  the  use  for  human  subjects. 

Task  6B.4.  Conduct  Experiment. 

Task  6B.5.  Analyze  Data. 

Task  6B.6.  Write  Report. 


Resource  Requirements  and  Availability 
Computer  Systems 

The  relevant  computer  capabilities  of  each  of  the  participating  government  organizations  are 
shown  in  Appendix  A.  (The  ARI  computer  is  located  at  1ST.)  All  use  the  Silicon  Graphics 
Onyx  Reality  Engine  as  the  principle  component.  However  the  systems  are  configured 
differently. 
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Terrain  Databases 


The  following  terrain  databases  are  required  for  the  assessment. 

Quantico  Combat  Training  Village.  NAWC-TSD  has  developed  this  database. 

Open  Terrain  Databases.  ARI  has  developed  two  two-kilometer  square  rural  terrain 
databases.  One  represents  no  real  area.  The  second  roughly  represents  a  section  of  the 
McKenna  MOUT  site  at  Fort  Benning,  GA. 

Airbase.  AL/HR  is  developing  this  database. 

K-D  Range.  ARL/HRED  is  developing  this  model  of  the  K-D  Range,  an  actual  obstacle 
course  located  at  Aberdeen  Proving  Ground,  MD. 

Cross-Country  Course.  ARL/HRED  is  developing  this  model  of  an  actual  cross  country 
course  located  at  Aberdeen  Proving  Ground,  MD. 

Cost  Rationale 

A  cost  estimate  for  the  assessment  is  presented  in  Appendix  B.  This  should  be  considered  a 
rough  estimate.  It  is  not  based  on  either  detailed  analysis  by  the  participating  organizations  or 
the  bids  from  contractors.  The  work  to  accomplish  each  task  has  been  allocated  between  the 
government  organization  (NAWC-TSD,  ARL/HRED,  etc.)  and  a  contractor,  either  a  university 
(University  of  Central  Florida,  University  of  Peimsylvania)  or  a  commercial  firm  (Sarcos 
Research,  Boston  Dynamics,  Inc.).  This  is  not  intended  to  indicate  how  the  work  should  be 
accomplished  but  rather  only  to  provide  a  basis  for  cost  estimation. 

Two  different  man-hour  costs  were  used.  A  staff  year  of  1 840  hours  was  used.  Government 
and  University  costs  were  based  of  the  current  NAWC-TSD  of  $90,000  per  staff  year,  or  $48.91 
per  hour.  Commercial  costs  were  based  on  a  burdened  cost  (including  indirect  cost  and 
overhead)  of  $130,000  per  year,  or  $70.65  per  hour. 

Travel  costs  were  based  on  current  government  airfares  and  per  diem  rates. 
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BOOM  2C;  various 

_ monitors _ 

What  type  of  position  tracking  is  Polhemus  Fastrac  (4  Flock  of  Birds  (8  Polhemus  Fastrak  (4 

available?  How  many  sensors?  sensors)  sensors)  sensors);  Flock  of  Birds 

_ _ (6  sensors) _ 


Appendix  A.  Computer  Resources 
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ARI 

IRIX  5.3 

2(1  with  degredation) 

20MB +  5  MB  per 
trainee 

128MB  minimum  I 

IRIS  Performer  1,2 

14.1  (not  required  to 
run) 

o 

00 

1 

•a 

o 

IS 

o 

ts 

2  ^ 
a  X 

Flock  of  Birds  (4 
sensors)  for  current 
gesture  set. 

No 

Special  chord-type 
glove,  voice  recognition 
card  (Dragon  Speak),  a 
486  PC  w/  networking 
capability,30  MB  of 
disk  space,DOS  6.22 

NAWCTSD 

IRIX5.3 

Not  cuirently  used 

50MB  with  one 
database  and  one  trainee 

64  MB 

IRIS  Performer  2.0 

G  ^ 

8  2 

1  i 

a  ^ 
s'  ^ 

&  s 

e3  § 

(2  e 

Flock  of  Birds  (1) 

No 

Custom  made  weapons 
interface 

ALHRG 

IRIX  5.2  or  higher 

o 

Not  currently  used 

1  recommended 

500  MB  recommended 
free  space 

32  MB  minimum 

IRIS  Performer  1.2 

Not  currently  required 

Graphics  Monitor,  1280 
xl024 

Hock  of  Birds  (8), 
CyberGlove  (or 
Dataglove),  Crystal 

Eves 

o 

iz; 

24-bit  Z-buffered 
graphics.  An  Indigo  2 
XZ  is  good  enough  for 
some  of  the  user’s 
applications. 

Requirements  for  Organization-Specific  Applications: 

ARL-HRED 

Minimum  2 

N/A 

< 

2 

200  MB 

64  MB 

IRIS  Performer  1.2 

N/A 

Standard  SGI  Graphics 
Monitor 

_  ..  .... 

Polhemus  Fastrak  (2 
sensors) 

NPSNET,  distributed 
free  by  Naval 
Postgraduate  School 

1  Version  of  IRIX  OS 

1  Number  of  processors 

1  Number  of  Serial  Ports  Required 

iMulti-Channel  Option  (MCO)  ? 

1  Number  of  Raster  Managers 

Disk  Space  Required 

1  Amount  of  Memory 

llRIS  Performer  toolkit  version  # 

MultiGen  version  # 

What  type  of  visual  display  systems  are 
required? 

What  type  of  position  tracking  is  used? 
How  many  sensors? 

Any  software  required  besides  IRIS 
and  Performer? 

Any  especially  configured  hardware 
required? 
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less  Assessment  Cost  Estimate:  Travel  Costs 


Airfares 


Orlando  to  Washington,  DC 

$398.00 

Philadelphia,  PA  to  Dayton,  OH 

$338.00 

Orlando  to  Philadelphia,  PA 

$402.00 

Orlando  to  Boston,  MA 

$342.00 

Orlando  to  Salt  Lake  City,  UT 

$630.00 

Orlando  to  Baltimore,  MD 

$364.00 

Per  Diem  Rates 

Room 

Meals 

Total 

Quantico,  VA 

$55.00 

$26.00 

$81.00 

Dayton,  OH 

$63.00 

$20.00 

$91.00 

Philadelphia,  PA 

$89.00 

$34.00 

$123.00 

Orlando.  FL 

$68.00 

$26.00 

$94.00 

lA.  Phila  to  Orlando.  1  person,  5  days,  5  trips 

$5,360.00 

IB  Phila  to  Dayton,  1  person,  5  days,  2  trips 

$1,986.00 

2.1  No  travel 

2.2  No  travel 

4.1  Orlando  to  Quantico,  3  persons,  5  days,  3  trips 

$5,739.00 

4.2  Quantico  to  Orlando,  1  person,  2  days,  4  trips 

$2,664.00 

5  No  travel 
6A.1  No  travel 
6A2.  No  travel 


6B.1  Orlando  to  Quantico,  3  persons,  5  days,  2  trips 

$3,706.00 

Lessons  Learned 

Baltimore  to  Orlando,  1  person,  2  days,  1  trip 

$632.00 

Philadelphia  to  Orlando,  2  persons,  2  days,  1  trip 

$1,280.00 

Quantico  to  Orlando,  Iperson,  2  days,  1  trip 

$666.00 

Boston  to  Orlando,  1  person,  2  days,  1  trip 

$610.00 

Dayton  to  Orlando,  1  person,  2  days,  1  trip 

$606.00 

Salt  Lake  City  to  Orlando,  1  person,  2  days,  1  trip 

$898.00 

Total  Lessons  Learned 

$4,692.00 

Total  Travel 

$24,147.00 
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APPENDIX  C:  LIST  OF  ACRONYMS 


AL/HR . US  Air  Force  Armstrong  Laboratory,  Human  Resource  Directorate 

ARI . US  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

ARL-HRED . US  Army  Research  Laboratory,  Human  Research  and  Engineering 

Directorate 

CCHC . Computer  Controlled  Hostile  Combatants 

CCHC/N . Computer  Controlled  Hostile  Combatants  and  Neutrals 

CGF . Computer  Generated  Forces 

CGFT . Computer  Generated  Forces  Testbed 

DIS . Distributed  Interactive  Simulation 

DOD . Department  of  Defense 

DOF . Degrees  of  Freedom 

less . Individual  Combatant  Simulation  System 

ISMS  . Individual  Soldier  Mobility  Simulator 

1ST . University  of  Central  Florida  Institute  for  Simulation  and  Training 

MARCORSYSCOM  .  US  Marine  Corps  Systems  Command 

MILES . Multiple  Integrated  Laser  Engagement  System 

MOBA . Military  Operations  in  Built-up  Areas 

MOUT . Military  Operations  in  Urban  Terrain 

NAWC-TSD  . US  Naval  Air  Warfare  Center  Training  Systems  Division 

PDP  . Program  Development  Plan 

PDU . Protocol  Data  Unit 

ROTC . Reserve  Officers  Training  Corps 

SAFOR . Semi-automated  Forces 

SME . Subject  Matter  Expert 

SSQ  . Simulator  Sickness  Questionnaire 

TTES  . Team  Target  Engagement  Simulator 

VE  . . . Virtual  Environment 

VR . Virtual  Reality 
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